Discrete Time Markov Chain (DTMC)

A. A stochastic process is a collection of random variables {X t , t ∈ T }. B. A sample path or realization of a stochastic process is the collection of values assumed by the random variables in one realization of the random process, e.g. C. The state space is the collection of all possible values the random variables can take on, i.e. it is the sample space of the random variables. For example, if X i ∈ [0, ∞) represent random times for all i, then the state space of the stochastic process is [0, ∞). D. Often, the index set T is associated with time, sometimes even when it does not actually represent time. In this description, the stochastic process has a state that evolves in time. For example, the process may start in state X 1 = 3, then evolve to state X 2 = 4, and much later enters state X 100 = 340. The index set may also be associated with space, for example T = 2 for the real plane. E. Classifying stochastic processes. Stochastic processes can be classified by whether the index set and state space are discrete or continuous. State Space discrete continuous Index discrete discrete time Markov chain (dtmc) not covered Set continuous continuous time Markov chain (ctmc) diffusion processes 1. Random variables of a discrete time process are commonly written X n , where n = 0, 1, 2,. . .. 2. Random variables of a continuous time process are commonly written X(t), where t ∈ T , and T is often, though certainly not always [0, ∞). F. Short history of stochastic processes illustrating close connection with physical processes.